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Abstract 

Decision  Analysis  (DA)  is  a  useful  tool  to  assist  decision  makers  (DM)  with 
difficult  and  complex  decisions  using  mathematical  models.  Value  Focused  Thinking 
(VFT)  models  are  a  useful  DA  tool  widely  employed  in  the  Air  Force.  However,  VFT 
models  are  rarely  validated. 

This  research  will  attempt  to  validate  any  given  VFT  model  and  provide  insight 
into  the  discriminating  attributes  of  the  alternative  set.  First,  a  two  group  discriminant 
analysis  is  applied  the  alternative  set  given  the  prior  knowledge  of  the  selected 
alternatives.  Next,  compromise  programming  is  used  attempt  to  minimize  the  distance 
between  the  posterior  probability  of  an  alternative  being  selected  and  its  current  weighted 
value  by  varying  the  weights.  This  set  of  optimized  weights  is  then  used  in  the  two  group 
discriminant  analysis  to  classify  the  alternative  set  and  attempt  to  validate  the  VFT  model 
by  selecting  the  same  subset  of  alternatives  chosen  by  the  DM.  Additionally,  this  process 
will  provide  insight  into  what  attributes  of  a  given  alternative  set  are  actually  the 
discriminating  factors  in  the  decision  which  may  or  may  not  be  the  attributes  that  are 
most  important  to  or  most  heavily  weighted  by  the  DM. 
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VALIDATION  OF  VALUE  FOCUSED  THINKING  DECISION  MODEES  USING 

MUETIVARIATE  TECHNIQUES 

I.  Introduction 

Air  Force  decision  makers  (DM)  are  forced  to  make  difficult  decisions  that  can 
have  significant  impact  to  future  force  capabilities  and  structure.  Often  these  decisions 
are  made  more  difficult  by  the  fact  that  the  goals  of  the  decision  are  often  conflicting. 
Decision  Analysis  (DA),  specific  to  this  research  Value  Focused  Thinking  (VET), 
provides  a  tool  that  allows  analysts  to  help  the  DM  make  tradeoffs  between  conflicting 
goals.  A  DM  needs  to  have  a  detailed,  robust  and  mathematically  rigorous  model  to  aid 
in  their  decision  making. 

I,A  Background 

Decision  makers  and  analysts  often  employ  VET  models  to  assist  in  making 
difficult  decisions.  The  VET  methodology  is  well  documented  and  often  used  in  today’s 
Air  Force.  The  VET  process  generates  a  VET  model  that  should  encompass  the  values 
and  objectives  of  the  DM  and  any  stakeholders.  Once  the  VET  hierarchy  is  completed, 
attributes  or  measures  are  weighted;  alternatives  are  scored,  evaluated,  and  then  ranked. 
Often  a  selection  threshold  is  set  and  any  alternatives  with  a  value  score  from  the  VET 
model  greater  than  the  threshold  are  selected.  Instead  of  a  value  score  threshold,  a  budget 
constraint  may  be  applied.  The  top  value  score  alternatives  up  to  a  resource  or  budget 
constraint  would  be  selected  in  this  case. 
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Alternatively,  the  VFT  results  may  not  refleet  the  final  deeision  and  some 
alternatives  that  scored  poorly  in  the  VFT  model  may  end  up  being  selected  based  upon 
other  factors  that  were  not  or  could  not  be  reflected  in  the  VFT  model.  In  a  case  such  as 
this,  there  may  be  a  logical  set  of  weights  that  can  be  applied  to  the  VFT  model  to 
account  for  the  DM’s  change  in  values  to  reflect  the  decision  outcome.  In  this  case,  the 
DM  may  be  weighting  an  attribute  that  he  feels  is  important  but  due  to  the  value  scores  of 
the  alternative  set  the  attribute  has  little  discriminatory  power.  For  example,  if  a  set  of 
alternatives  all  scored  very  well  in  the  most  important  attribute  to  the  DM,  but  varied 
greatly  in  an  attribute  that  was  less  important,  through  discriminant  analysis  we  can 
provide  insight  into  whether  that  less  important  attribute  is  actually  having  the  most 
influence  in  making  the  decision. 

I.B  Problem  Statement 

Decision  makers  often  have  to  make  complex  and  difficult  decisions.  Ensuring 
decision  analysis  techniques  such  as  VFT  are  accurately  applied  and  provide  valid, 
repeatable  and  accurate  results  helps  to  justify  allocation  of  scarce  resources  to  the  many 
areas  they  are  needed.  This  research  provides  a  framework  to  help  ensure  consistent 
decision  analysis  techniques  are  used  and  to  help  provide  insights  as  a  result  of  the 
decision  that  may  not  be  readily  apparent. 

I,C  Research  Scope 

This  thesis  will  evaluate  the  validity  of  a  given  VFT  model.  We  will  address 
whether  the  VFT  model  accurately  reflects  the  DM’s  decision  process  in  light  of  the 
alternatives  presented  him  using  discriminant  analysis  with  a  two  group  problem.  After 
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this  validation  process  we  provide  insight  to  the  DM  and  stakeholders  by  determining 
whieh  attributes  of  the  alternatives  had  the  most  influence  on  whether  they  were  seleeted 
or  not  seleeted. 

I.D  Assumptions 

This  research  does  not  require  any  speeific  assumptions  regarding  the  VFT  model 
or  the  alternative  set. 

I,E  Thesis  Organization 

The  remainder  of  this  thesis  is  eomposed  of  four  chapters.  Chapter  2  eonsists  of  a 
literature  review  of  DA,  VFT,  weighting  teehniques,  sensitivity  analysis,  multivariate 
analysis,  and  diseriminant  analysis  teehniques.  Chapter  3  outlines  the  methodology  used 
by  this  researeh  including  discriminant  analysis  modeling,  and  compromise  programming 
optimization  of  the  diseriminant  weights.  Chapter  4  consists  of  the  results  of  the 
applieation  of  the  methodologies  covered  in  chapter  3  when  applied  to  alternative  sets 
and  VFT  models.  Finally,  ehapter  5  discusses  the  conclusions  and  suggests  opportunities 
for  future  research. 
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II.  Literature  Review 


II.A  Introduction 

The  literature  review  for  this  work  eneompasses  relevant  material  regarding 
Deeision  Analysis  (DA)  and  value  models.  We  will  discuss  the  fundamentals  of  those 
disciplines  along  with  some  of  the  multivariate  analysis  techniques  that  are  pertinent  to 
this  research. 

II.B  Decision  Analysis 

Decision  Analysis  (DA)  is  an  analysis  tool  that  allows  a  Decision  Maker  (DM)  to 
have  a  repeatable,  mathematically  rigorous  process  to  aid  in  making  difficult  decisions. 
“DA  provides  structure  and  guidance  for  thinking  systematically  about  hard  decisions.” 
(Clemen  &  Reilly,  2001).  As  described  by  Clemen,  we  must  start  by  identifying  the  DM 
or  DMs.  Next,  the  DA  process  begins  by  identifying  the  decision  situation  and  ensuring 
we  understand  the  objectives.  We  then  identify  the  alternatives  to  be  considered.  Next, 
follows  the  decomposition  and  modelling  of  the  problem.  The  VFT  model  will  show  us 
how  the  alternatives  faired  and  we  can  choose  the  best  alternative  or  if  necessary  try  to 
generate  new  alternatives  if  those  provided  did  not  meet  objectives  to  the  levels  required 
by  the  DM.  We  can  also  conduct  sensitivity  analysis  on  the  results  of  the  model  and 
determine  if  further  analysis  is  needed.  Finally  if  no  further  analysis  is  required  we  can 
implement  our  chosen  alternative.  DA  is  not  intended  to  replace  the  DM  or  make  the 
decision  for  them.  DA  is  a  useful  tool  to  provide  insights  and  help  the  DM  make  an 
informed  decision. 
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II.C  Value  Focused  Thinking 


The  methods  derived  in  from  this  researeh  ean  be  applied  to  any  Value  Focused 
Thinking  (VFT)  model  that  is  used  to  evaluate  alternatives  and  then  select  some  subset  of 
the  alternatives  to  be  selected  for  funding  or  some  other  function.  VFT  techniques  differ 
from  alternative  based  techniques  since  the  latter  approach  considers  the  best  of  the 
available  alternatives  (Keeney,  1992).  A  VFT  approach  shows  how  much  value  is 
provided  by  an  alternative  and  can  lead  to  development  of  new  alternatives  based  upon 
revelations  provided  by  the  value  model. 

Problem  identification  is  an  important  first  step  in  the  VFT  process.  It  is  essential 
to  make  sure  you  are  looking  at  the  right  problem.  Discussions  with  the  DM  and 
stakeholders  provide  clarity  with  regards  to  the  problem  identification.  The  DM’s  values 
are  determined  by  what  is  important  to  him  or  her.  It  is  sometimes  useful  to  consider 
strategic  plans  and  objectives  that  have  been  developed  by  the  DM.  They  can  usually 
provide  insight  into  the  DM’s  values  and  aid  in  development  of  better  alternatives. 
Weights  are  assigned  to  the  value  attributes  using  one  of  various  weighting  techniques 
and  based  upon  the  importance  of  the  value  to  the  decision  maker.  The  more  important 
the  attribute’s  value  is  to  the  DM,  the  more  weight  it  will  have  in  the  model.  Alternatives 
are  then  considered  and  scored  based  upon  value  functions  for  the  individual  value 
measures.  The  resulting  VFT  model  output  will  give  an  overall  value  score  for  each 
alternative.  This  will  provide  insight  into  which  alternative  will  give  the  most  value 
based  upon  the  DM’s  preferences.  Sensitivity  analysis  would  then  be  performed  to  check 
the  robustness  of  the  VFT  model. 
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II.D  VFT  Weighting  Techniques 


There  are  several  weighting  teehniques  available  for  use  in  VFT  models  and  this 
researeh  is  applieable  to  any  weighting  teehnique.  Poyhonen  deseribes  some  weighting 
teehniques  ineluding  Direet  weighting,  Simple  Multiattribute  Rating  Teehnique 
(SMART),  Swing  weighting.  Tradeoff  weighting,  and  the  Analytie  Hierarehy  Proeess 
(AHP)  (2001). 

In  Direet  weighting,  the  DM  just  assigns  weights  to  eaeh  attribute.  Typieally  the 
weights  are  assigned  such  that  they  sum  to  one.  This  is  not  necessary  but  it  is  usually 
easier  for  the  DM  and  stakeholders  to  relate  to  the  meaning  of  the  weight  if  it  is  scaled  on 
a  zero  to  one  scale.  Assigning  weights  with  the  SMART  technique  involves  ranking  the 
importance  of  changes  in  attributes  when  an  attribute  moves  from  its  lowest  score  to  its 
highest  score.  Ratio  estimates  are  assigned  sequentially  starting  with  the  least  important 
attribute.  In  SWING  weighting  the  DM  is  given  a  hypothetical  alternative  that  has  the 
worst  score  for  each  attribute.  He  then  chooses  his  most  important  attribute  to  increase  to 
its  highest  level  and  assigns  100  points  to  it.  The  next  most  important  attribute  is  then 
chosen  to  increase  to  its  highest  level  and  assigned  something  less  than  100  points.  This 
process  continues  until  all  attributes  are  assigned  a  weight.  In  the  Tradeoff  method  the 
DM  is  asked  to  compare  2  hypothetical  alternatives  that  differ  in  two  attribute  measures 
only  with  all  other  attribute  scores  held  constant.  The  DM  then  compares  the  alternatives 
and  is  asked  to  change  the  attribute  scores  until  he  is  indifferent  between  the  two 
alternatives.  That  is,  he  likes  both  alternatives  equally  as  well  even  though  they  have  2 
attributes  that  are  not  the  same.  This  would  have  to  be  done  n-1  times  to  get  all  the 
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indifference  statements,  where  n  is  the  number  of  attributes.  The  indifference  statements 
are  used  to  generate  equations  relating  the  respective  weights  and  value  functions  for  the 
attributes.  Those  n-1  equations  plus  a  normalization  equation  allow  calculation  of  the 
weights.  In  AHP  the  DM  is  asked  to  compare  two  attributes  at  a  time  and  give  a  relative 
importance  between  the  two  for  each  combination  of  two  weights.  The  relative 
importance  is  determined  by  a  weight  ratio  assigned  by  the  DM  for  each  pair  of 
attributes.  The  weight  ratio  is  typically  an  integer  from  1  to  9.  Once  this  is 
accomplished  for  all  pair  combinations  of  attributes,  the  weights  are  usually  derived  from 
the  principle  eigenvector  of  the  comparison  matrix  (Poyhonen  &  Hamalainen,  2001). 

II.E  Sensitivity  Analysis 

Often  there  is  uncertainty  in  many  key  parts  of  the  DA  process  and  we  need  to 
conduct  analysis  on  the  results  of  the  VFT  model  with  regards  to  this  uncertainty. 
Typically  this  is  done  by  varying  the  weights.  This  allows  us  to  look  for  changes  in  the 
rankings  of  the  alternatives  as  the  weights  are  varied.  If  the  rankings  change  with  small 
changes  in  the  weights  then  the  decision  is  said  to  be  sensitive  to  changes  in  that  weight 
or  weights.  If  the  rankings  don’t  change  or  only  change  with  proportionally  large 
variation  in  the  weights  then  the  decision  is  insensitive  to  the  weight  or  weights. 

Clemen  and  Reilly  discuss  methods  for  one-way  and  two-way  sensitivity  analysis 
where  one  (or  two)  weights  are  varied  while  all  other  weights  are  kept  proportionate  to 
their  original  weights  (Clemen  &  Reilly,  2001). 

Bauer,  Parnell,  and  Meyers  present  a  method  using  Response  Surface 
Methodology  to  perform  higher  order  sensitivity  analysis.  The  value  functions  from  the 
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VFT  model  are  transformed  into  a  response  function  of  the  uncertain  variables.  RSM 
was  used  to  create  a  sensitivity  analysis  framework  that  allowed  simultaneous 
perturbation  of  a  number  of  uncertain  variables  (Bauer  et  al,  1999). 

Ringuest  presents  a  methodology  where  the  Li  and  Loo  metrics  are  minimized 
subject  to  linear  constraints  (Ringuest,  1997).  He  suggests  that  since  the  constraints  are 
linear,  solving  the  linear  program  minimizing  the  Li  and  Loo  metrics  will  completely 
specify  the  solutions  which  minimize  the  Lp  metric.  The  Lp  metric  is  the  generalized  form 
for  P  effect  on  the  relative  contribution  of  individual  deviations  (Ringuest,  1997). 

II,F  Multivariate  Analysis 

Dillon  and  Goldstein  define  Multivariate  analysis  as  the  application  of  methods 
that  deal  with  large  numbers  of  measurements  made  on  each  object  in  one  or  more 
samples  simultaneously  (1984).  The  simultaneous  aspect  of  the  multiple  variables  and 
the  analysis  of  all  the  variables  at  once  instead  of  one-way  analysis  is  the  key  point  to 
multivariate  analysis 

II.G  Discriminant  Analysis 

Discriminant  analysis  is  a  statistical  technique  for  classifying  individuals  or 
objects  into  mutually  exclusive  and  exhaustive  groups  on  the  basis  of  a  set  of  independent 
variables  (Dillon  &  Goldstein,  1984).  This  is  done  by  deriving  a  linear  combination  of 
the  independent  variables  that  determines  the  difference  between  the  a  priori  defined 
groups  so  that  misclassification  is  minimized.  The  technique  attempts  to  maximize  the 
between  group  variance  relative  to  within  group  variance.  The  linear  combination  is 
determined  by  a  weighted  average  of  each  object  or  individual’s  scores  on  the 
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independent  variables.  This  is  then  turned  into  a  posteriori  probability  of  being  assigned 
to  eaeh  group.  How  well  the  diseriminant  funetion  performs  is  determined  by  how  well 
the  funetion  elassifies  the  objeets  or  individuals. 

II.H  Compromise  Programming 

Compromise  programming  is  a  speeifie  form  of  multieriteria  optimization 
programming  that  attempts  to  minimize  the  distanee  from  an  objeetive  or  ideal  point  to 
the  alternative  spaee.  The  general  form  is  shown  below. 

Given  a  distanee  measure  ,  the  eompromise  programming 

problem  is  given  by  ,  where  is  the  ideal  point  and  any 

appropriate  distanee  measure  as  a  funetion  of  (Ehrgott,  2005).  Typieally,  norms  sueh 
as  the  norm  or  sums  of  square  differenee  are  used  as  the  distanee  measure. 

II.I  Summary 

This  ehapter  presents  a  review  of  literature  that  provided  key  elements  of  DA, 
VFT,  weighting  teehniques,  and  sensitivity  analysis.  A  vital  part  of  this  researeh  eenters 
on  the  Multivariate  analysis  teehnique  of  Diseriminant  Analysis.  Uneertainty  inherent  to 
diffieult  deeisions  will  allow  us  to  explore  applieation  of  Diseriminant  Analysis 
teehniques  to  assess  sensitivity  of  VFT  models  with  regards  to  a  given  alternative  set. 
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Ill,  Methodology 


III.A  Introduction 

The  purpose  of  this  ehapter  is  to  present  the  methods  used  to  assess  the  robustness 
of  a  VFT  model’s  grouping  of  seleeted  and  non-selected  alternatives  based  on  a  given  set 
of  alternatives  and  global  weights.  This  research  will  attempt  to  find  the  best  set  of 
global  weights  that  will  generate  the  same  selection  grouping  of  alternatives  generated  by 
the  VFT  model.  First,  this  research  uses  discriminant  analysis  to  determine  if  a 
discriminant  function  can  reasonably  classify  the  alternative  set  into  the  selected  and  non- 
selected  groups.  Next,  the  posterior  probabilities  generated  by  the  discriminant  analysis 
are  used  along  with  the  value  scores  of  the  alternatives  to  create  a  math  programming 
model  to  find  the  optimal  set  of  weights  that  produce  alternative  value  assessments  that 
are  as  close  as  possible  to  the  posterior  probability  estimates. 


III.B  VFT  Model 

This  research  assumes  that  a  given  VFT  model  was  properly  formulated  and  in 
the  form  shown  in  equation  1 ,  weightings  were  chosen  by  the  DM  or  analysts  in  an 
appropriate  fashion  and  the  results  of  the  VFT  model  were  attained. 


(1) 


where: 

:  the  evaluation  measures  for  alternative  j  for  each  of  the  i  attributes 
:  the  weight  associated  with  attribute  i. 

:  the  individual  value  functions  associated  with  attribute  i 
:  the  overall  value  score  of  alternative  j 
N:  the  number  of  attributes,  i=\  to  N 
n:  the  number  of  alternatives,  7=1  to  n 
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The  alternatives  ean  then  be  ordered  based  upon  their  value  seore 


such  that 


.  Let  T  be  a  threshold  of  the  value  score  that 
divides  the  alternatives  into  2  sets.  These  sets  represent  the  alternatives  that  were 
selected  and  rejected  based  on  the  threshold,  T.  It  is  possible  that  the  sets  are  constructed 
in  some  other  manner  independent  of  the  value  model. 

III.C  Discriminant  Analysis 

Discriminant  analysis  is  “a  statistical  technique  for  classifying  individuals  or 
objects  into  mutually  exclusive  and  exhaustive  groups  on  the  basis  of  a  set  of  independent 
variables”  (Dillon  &  Goldstein,  1984).  Discriminant  analysis  tries  to  find  a  linear 
combination  of  the  independent  variables  that  will  discriminate  between  the  a  priori 
defined  groups  while  minimizing  misclassification  error.  This  is  accomplished  by 
maximizing  the  between-group  variance  relative  to  the  within-group  variance  (Dillon  & 
Goldstein,  1984). 

Discriminant  analysis  uses  a  scoring  system  that  assigns  each  object  a  score  that  is 
a  weighted  average  of  the  object’s  values  on  the  set  of  independent  variables.  This 
discriminant  score  is  then  transformed  into  an  a  posteriori  probability  that  assigns  a 
likelihood  of  the  object  belonging  to  each  of  the  groups.  In  this  research,  the  a  priori 
groups  are  defined  by  VFT  results  and  any  supplemental  analysis  used  by  the  DM.  The 
independent  variables  used  in  the  discriminant  analysis  are  the  value  function  scores  of  a 
given  alternative  for  each  attribute  assessed  in  the  VFT  model.  The  idea  is  to  model  the 
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alternatives  evaluated  in  the  VFT  model  as  multivariate  vectors  of  their  individual  value 


function  assessments. 

The  equation  used  to  generate  the  vector  of  discriminant  weights  using  Fisher’s 
Approach  to  a  two  group  problem  is  shown  in  equation  2  below  (Dillon  &  Goldstein, 
1984): 


(2) 


where: 

:  the  vector  of  discriminant  weights 

:  the  pooled  sample  variance-covariance  matrix. 

:  the  mean  of  the  observations  in  the  i*  population 

The  discriminant  scores  are  then  created  by  equation  3  below. 


(3) 


where: 

:  the  vector  of  discriminant  scores 

:  the  matrix  of  alternative  value  scores  from  VFT  model  where  each  row  is  an 
alternative  and  the  columns  are  the  attribute  value  scores 
:  the  vector  of  discriminant  weights 

Once  the  discriminant  scores  are  obtained,  classification  is  made  by  comparing 
the  discriminant  score  to  a  midpoint  between  the  means  of  the  two  population  groups. 

This  midpoint  is  calculated  using  equation  4  below.  A  discriminant  score  greater  than  the 
midpoint  would  be  assigned  to  group  1  and  less  than  would  be  assigned  to  group  2  (Bauer 
K.  W.,2010): 


(4) 
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Calculation  of  the  a  posteriori  probability  is  now  necessary  by  using  the 


following  equation  5. 

_  _  ~  ^  (5) 

where: 

:  the  a  posteriori  probability  of  alternative  j  being  in  the  selected 

group 

III.D  Compromise  Programming 

Once  the  discriminant  analysis  has  calculated  the  a  posteriori  probability  of  each 
alternative  x,  we  use  that  posterior  probability  as  the  ideal  point  in  a  compromise 
programming  problem  that  minimizes  the  difference  between  the  value  of  the  alternative 
and  the  posterior  probability  of  being  selected.  The  thought  behind  this  is  that  an 
alternative  that  is  selected  should  have  a  high  VFT  value  score  and  also  a  high  posterior 
probability  of  being  in  the  selected  population.  Minimizing  the  difference  between  those 
numbers  should  increase  the  separation  between  the  selected  and  non-selected 
populations.  Equation  6  below  shows  the  compromise  programming  problem  with  the 
posterior  probability  of  the y'th  alternative  being  classified  in  the  selected  group  as  the 
ideal  point. 


(6) 


where: 

:  the  value  of  the  y'th  alternative  for  the  /th  attribute  where  i=  1,. .  .,N  and 
j=  l,...,n 
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:  the  weight  of  the  rth  attribute 


III.E  Validation  and  Analysis 

Once  the  new  weight  vector  is  obtained  from  the  compromise  programming,  the 
classification  can  be  tested  using  the  two  group  discriminant  analysis  method.  If  the  new 
discriminant  does  a  reasonable  job  of  classifying  the  alternatives  relative  to  the  original 
decision  than  we  can  conclude  that  the  decision  being  made  is  statistically  consistent  and 
valid. 

Given  the  new  weights  we  may  have  some  new  insights  into  the  alternative  set. 
Any  attribute  weight  that  was  reduced  to  zero  or  near  zero  was  done  so  because  it  had  no 
discriminatory  power.  Any  attribute  weights  that  are  very  large  have  significant 
discriminatory  power.  This  will  provide  insight  into  which  attributes  actually  determined 
which  alternatives  were  selected  and  may  or  may  not  reflect  the  weights  determined  from 
the  VFT  process. 

III.F  Summary 

This  chapter  consisted  of  the  methodology  used  in  this  research  to  validate  VFT 
models  and  provide  insight  into  a  decision  based  upon  a  given  alternative  set.  This 
research  uses  a  given  alternative  set  that  has  been  processed  through  a  VFT  model  and 
the  selection  outcome  determined  by  a  VFT  value  score  threshold  or  some  other  method. 
Then,  two  group  discriminant  analysis  techniques  and  compromise  programming 
methods  are  applied.  This  method  shows  that  statistically  there  is  a  set  of  weights  that 
will  produce  the  same  selected  and  non-selected  group  and  provide  insights  into  what 
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attributes  of  the  alternatives  are  the  diseriminating  factors.  A  copy  of  the  MATLAB  code 
utilized  during  this  research  is  includes  in  Appendix  A. 
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IV,  Results  and  Analysis 


IV.A  Introduction 

This  chapter  applies  the  methodologies  diseussed  in  ehapter  3  to  a  VFT  model 
and  alternatives  of  an  unspecified  Air  Foree  Major  Command  (MAJCOM),  and  to  a 
JIEDDO  VFT  model  and  set  of  alternatives  used  previously  in  Willy’s  thesis  (Willy, 
2009).  In  each  case  we  will  validate  the  deeision  by  using  the  diseriminant  analysis  and 
eompromise  programming  teehniques  identified  in  ehapter  3  to  duplieate  the  deeision 
using  statistieal  rigor.  Then,  for  eaeh  alternatives  set  we  will  utilize  the  weights 
generated  by  the  previous  proeess  to  see  if  any  insights  are  gained  by  the  results.  Finally, 
we  will  demonstrate  an  example  of  a  poor  hit  rate  from  an  ineonsistent  decision. 


IV.B  MAJCOM  Data 


The  set  of  MAJCOM  alternatives  is  shown  in  Appendix  B.  The  seleetion/non- 


seleetion  eutoff  thresholds  used  were  a  VFT  value  seore  of  0.5  and  above  for  the  seleeted 


population,  0.6  and  above,  0.7  and  above,  and  finally  0.8  and  above.  Initially  the  weights 
were  only  allowed  to  vary  by  ±10%  and  ±50%  of  their  original  value.  This  yielded 
weights  as  shown  in  Table  1  and  Table  2  below. 


Table  1  MAJCOM  Weights  for  ±10%  Bounds 


±10%  of  Original  Weights 

Selection 

Attribute  Weights 

Threshold  (#SeI/ 

2 

3 

4 

5 

6 

7 

8 

9 

10 

11 

12 

13 

14 

0.6  (30/42) 

0.066 

0.16 

0.088 

0.187 

0.153 

0.0108 

0.033 

0.0162 

0.066 

0.0724 

0.0738 

0.0296 

0.0221 

0.0221 

0.7  (25/42) 

0.066 

0.16 

0.088 

0.187 

0.153 

0.0108 

0.033 

0.0162 

0.066 

0.0724 

0.0738 

0.0296 

0.0221 

0.0221 

0.8  (12/42) 

0.066 

0.16 

0.088 

0.187 

0.153 

0.0108 

0.033 

0.0162 

0.066 

0.0724 

0.0738 

0.0296 

0.0221 

0.0221 

Original  Weights 

0.06 

0.17 

0.08 

0.17 

0.17 

0.012 

0.03 

0.018 

0.06 

0.06578 

0.08211 

0.03289 

0.02461 

0.02461 

I  Denotes  weight  set  to  Lower  bound  |  |  Denotes  weight  set  to  Upper  bound  | 
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Table  2  MAJCOM  Weights  for  ±50%  Bounds 


±50%  of  Original  Weights 

Selection 

Attribute  Weights 

Threshold  (#  Sel  / 

2 

3 

4 

5 

6 

7 

8 

9 

10 

11 

12 

13 

14 

0.6  (30/42) 

0.03 

0.085 

0.12 

0.255 

0.216 

0.006 

0.045 

0.027 

0.09 

0.0434 

0.0411 

0.0165 

0.0123 

0.0123 

0.7  (25/42) 

0.03 

0.085 

0.12 

0.255 

0.255 

0.008 

]  0.015 

0.027 

0.09 

0.0329 

0.0411 

0.0165 

0.0123 

0.0123 

0.8  (12/42) 

0.03 

0.085 

0.12 

0.255 

0.1262 

0.018 

0.045 

0.027 

0.09 

0.0329 

0.0477 

0.0493 

0.03692 

0.03692 

Original  Weights 

0.06 

0.17 

0.08 

0.17 

0.17 

0.012 

0.03 

0.018 

0.06 

0.06578 

0.08211 

0.03289 

0.02461 

0.02461 

I  Denotes  weight  set  to  Lower  bound  |  |  Denotes  weight  set  to  Upper  bound  | 


After  looking  at  the  results  and  noticing  that  several  statistical  weights  were  consistently 


set  to  their  upper  and  lower  bounds,  we  changed  the  bounds  to  allow  the  weights  to  vary 


from  0  to  1.  This  provided  some  interesting  results  shown  in  Table  3  below. 


Table  3  MAJCOM  Weights  for  0  to  1  Bounds 


0  to  1  Bound  for  Weights 

Selection 

Attribute  Weights 

Threshold  (#  Sel/ 

1 

2 

3 

4 

5 

6 

7 

8 

9 

10 

11 

12 

13 

14 

0.6  (30/42) 

0.085 

0 

0.136 

0.423 

0.129 

0 

0.026 

0.111 

0.082 

0.008 

0 

0 

0 

0 

0.7  (25/42) 

0 

0 

0.087 

0.322 

0.103 

0 

0 

0.253 

0.206 

0.029 

0 

0 

0 

0 

0.8  (12/42) 

0 

0 

0.101 

0.017 

0 

0 

0.007 

0 

0.467 

0 

0 

0.408 

0 

0 

Original  Weights 

0.06 

0.17 

0.08 

0.17 

0.17 

0.012 

0.03 

0.018 

0.06 

0.06578 

0.08211 

0.03289 

0.02461 

0.02461 

As  we  can  see,  there  were  several  statistical  weights  set  to  zero  and  significant 


portions  of  the  weight  assigned  to  only  a  few  attributes.  The  hit  rates,  shown  in  Table  4, 


were  excellent  for  the  resultant  discriminant  analysis.  This  statistically  validates  the 


MAJCOM  VFT  model.  We  were  able  to  use  a  statistical  method  in  discriminant  analysis 


to  verify  that  the  selection  grouping  developed  Ifom  the  VFT  model  were  consistent  and 


mathematically  sound. 
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Table  4  MAJCOM  Confusion  Matrices 


MAJCOM  DATA  ±10%  Bound 

0.6 

Threshold 

(30/42) 

Select 

Non  Select 

Select 

30 

0 

Non  Select 

0 

12 

Hit  Rate 

100.0% 

0.7 

Threshold 

(25/42) 

Select 

Non  Select 

Select 

25 

0 

Non  Select 

0 

17 

Hit  Rate 

100.0% 

0.8 

Threshold 

(12/42) 

Select 

Non  Select 

Select 

12 

0 

Non  Select 

1 

29 

Hit  Rate 

97.6% 

MAJCOM  DATA  ±50%  Bound 

0.6 

Threshold 

(30/42) 

Select 

Non  Select 

Select 

30 

0 

Non  Select 

0 

12 

Hit  Rate 

100.0% 

0.7 

Threshold 

(25/42) 

Select 

Non  Select 

Select 

25 

0 

Non  Select 

0 

17 

Hit  Rate 

100.0% 

0.8 

Threshold 

(12/42) 

Select 

Non  Select 

Select 

12 

0 

Non  Select 

1 

29 

Hit  Rate 

97.6% 

MAJCOM  DATA  0  to  1  Bound 

0.6 

Threshold 

(30/42) 

Select 

Non  Select 

Select 

30 

0 

Non  Select 

0 

12 

Hit  Rate 

100.0% 

0.7 

Threshold 

(25/42) 

Select 

Non  Select 

Select 

25 

0 

Non  Select 

0 

17 

Hit  Rate 

100.0% 

0.8 

Threshold 

(12/42) 

Select 

Non  Select 

Select 

12 

0 

Non  Select 

1 

29 

Hit  Rate 

97.6% 
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This  statistically  validates  the  MAJCOM  VFT  model.  We  were  able  to  use  a 
statistical  method  in  discriminant  analysis  to  verify  that  the  selection  grouping  developed 
from  the  VFT  model  were  consistent  and  mathematically  sound. 

In  trying  to  determine  how  the  discriminant  function  and  the  compromise 
programming  are  generating  the  statistical  weights  it  is  useful  to  consider  the  mean  of  the 
selected  group  when  compared  to  the  mean  of  the  non-selected  group  and  the  variance  of 
the  two  groups.  Table  5  shows  a  portion  of  this  data  with  the  full  results  in  Appendix  C. 
This  provides  insight  into  what  attribute  measures  are  actually  influencing  the  decision. 
Those  measures  with  statistical  weight  assigned  contribute  to  explaining  how  the 
alternatives  were  assigned  to  the  selected  and  non-selected  groups  and  the  measures  that 
were  assigned  zero  weight  either  were  not  a  discriminating  factor  or  had  too  much  in 
group  variation  to  be  useful. 

Table  5  MAJCOM  Sample  of  Means  and  Variances 


0.6  Selection 

Cutoff 

Measure  1 

Measure  2 

Measure  3 

Measure  4 

Measure  5 

Measure  6 

Selected  Mean 

0.950 

0.847 

0.610 

0.831 

0.780 

0.467 

Non-selected 

Mean 

0.813 

0.729 

0.125 

0.342 

0.575 

0.500 

Selected 

Variance 

0.015 

0.019 

0.180 

0.070 

0.024 

0.257 

Non-selected 

Variance 

0.092 

0.069 

0.051 

0.159 

0.091 

0.273 

0.7  Selection 

Cutoff 

Measure  1 

Measure  2 

Measure  3 

Measure  4 

Measure  5 

Measure  6 

Selected  Mean 

0.970 

0.860 

0.628 

0.847 

0.808 

0.520 

Non-selected 

Mean 

0.824 

0.744 

0.241 

0.461 

0.594 

0.412 

Selected 

Variance 

0.012 

0.019 

0.180 

0.079 

0.020 

0.260 

Non-selected 

Variance 

0.068 

0.052 

0.121 

0.151 

0.070 

0.257 

Denotes  a  weighted 
attribute  with  0  to  1  bound 

Denotes  attribute  with 
zero  weight  assigned 
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When  the  bounds  for  the  weights  are  relaxed  to  0  to  1 ,  weight  is  assigned  to  those 
measures  that  typically  have  the  greatest  difference  between  selected  and  non-selected 
means.  There  were  some  instances  where  there  was  a  significant  difference  in  means  but 
a  zero  weight  was  still  assigned.  This  usually  occurred  when  the  variances  of  the  two 
groups  was  large  enough  that  it  reduced  the  effectiveness  of  the  differences  in  the  means 
to  discriminate. 


IV.C  JIEDDO  Data 


The  same  techniques  were  applied  to  a  set  of  alternative  data  shown  in  Appendix 
D  and  value  model  from  JIEDDO  proposals  (Willy,  2009).  In  the  original  VET  model 
there  were  actually  13  measures.  However,  the  given  alternative  set  scored  exactly  the 
same  value  for  the  Training  Level  measure.  This  measure  was  removed  for  the  analysis 
portion  since  it  had  no  discriminatory  information.  Runs  were  conducted  using  value 
scores  of  0.4  and  above  for  the  selected  group,  0.5  and  above,  0.6  and  above,  and  finally 
the  actual  selected  group  decided  by  the  JIEDDO  DMs,  which  did  not  follow  the  VET 
model  ranking  exactly. 

The  results  were  similar  to  the  MAJCOM  data  in  that  there  were  only  a  few 
attributes  that  were  weighed  heavily  and  some  others  were  set  to  zero  as  shown  in  Table 
6  below. 


Table  6  JIEDDO  Weights  for  0  to  1  Bounds 


Selection 
Threshold  (#  Sel  / 

Attribute  Weights 

1 

2 

3 

4 

5 

6 

7 

8 

9 

10 

11 

12 

0.4  (25/30) 

0 

0.076 

0 

0 

0 

0 

0 

0 

0.001 

0.923 

0 

0 

0.5  (19/30) 

0 

0.189 

0 

0.171 

0 

0 

0.393 

0 

0 

0.247 

0 

0 

0.6  (7/30) 

0 

0.123 

0 

0.49 

0.051 

0 

0 

0.234 

0 

0 

0.102 

0 

JIEDDO  Selection 

0.041 

0.398 

0 

0 

0 

0 

0.134 

0.377 

0.05 

0 

0 

0 

Original  Weights 

0.056 

0.176 

0.056 

0.112 

0.11 

0.056 

0.091 

0.037 

0.056 

0.1 

0.087 

0.05 
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Similar  to  the  MAJCOM  data  results,  weight  was  distributed  aeross  only  a  few  of 
the  attributes.  Even  with  sueh  a  weight  distribution  we  still  had  an  exeellent 
elassifieation  hit  rate  as  shown  in  Table  7.  This  statistieally  validates  the  JIEDDO  VET 
model.  We  were  able  to  use  a  statistieal  method  in  diseriminant  analysis  to  verify  that  the 
seleetion  grouping  developed  from  the  VET  model  were  eonsistent  and  mathematieally 
sound. 
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Table  7  JIEDDO  Confusion  Matrices 


JIEDDO  DATA  ±10%  Bound 


0.4 

Threshold 

(25/30) 

Select 

Non  Select 

Select 

25 

0 

Non  Select 

0 

5 

Hit  Rate 

100.0% 

0.5 

Threshold 

(19/30) 

Select 

Non  Select 

Select 

19 

0 

Non  Select 

0 

11 

Hit  Rate 

100.0% 

0.6 

Threshold 

(7/30) 

Select 

Non  Select 

Select 

7 

0 

Non  Select 

1 

22 

Hit  Rate 

96.7% 

JIEDDO 

Select 

Non  Select 

Select 

14 

1 

Selection 

Non  Select 

1 

14 

Hit  Rate 

93.3% 

JIEDDO  DATA  ±50%  Bound 

0.4 

Select 

Non  Select 

Select 

25 

0 

(25/30) 

Non  Select 

0 

5 

Hit  Rate 

100.0% 

0.5 

Threshold 

(19/30) 

Select 

Non  Select 

Select 

19 

0 

Non  Select 

0 

11 

Hit  Rate 

100.0% 

0.6 

Threshold 

(7/30) 

Select 

Non  Select 

Select 

7 

0 

Non  Select 

1 

22 

Hit  Rate 

96.7% 

JIEDDO 

Select 

Non  Select 

Select 

14 

1 

Selection 

Non  Select 

1 

14 

Hit  Rate 

93.3% 

JIEDDO  DATA  0  to  1  Bound 


0.4 

Threshold 

(25/30) 

Select 

Non  Select 

Select 

24 

1 

Non  Select 

0 

5 

Hit  Rate 

96.7% 

0.5 

Select 

Non  Select 

Select 

19 

0 

(19/30) 

Non  Select 

0 

11 

Hit  Rate 

100.0% 

0.6 

Threshold 

(7/30) 

Select 

Non  Select 

Select 

7 

0 

Non  Select 

1 

22 

Hit  Rate 

96.7% 

Select 

Non  Select 

JIEDDO 

Select 

14 

1 

Selection 

Non  Select 

1 

14 

Hit  Rate 

93.3% 
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Comparing  the  mean  of  the  selected  group  to  the  mean  of  the  non-selected  group 
and  the  variance  of  the  two  groups  yields  similar  results  as  the  MAJCOM  data.  Table  8 
shows  a  portion  of  this  data  with  the  full  results  in  Appendix  E. 

Table  8  JIEDDO  Sample  of  Means  and  Variances 


0.4  Selection 

Cutoff 

Measure  1 

Measure  2 

Measure  3 

Measure  4 

Measure  5 

Measure  6 

Selected 

Mean 

0.530 

0.510 

0.730 

0.540 

0.310 

0.750 

Non-selected 

Mean 

0.550 

0.000 

0.750 

0.452 

0.250 

0.700 

Selected 

Variance 

0.064 

0.166 

0.005 

0.193 

0.048 

0.068 

Non-selected 

Variance 

0.013 

0.000 

0.000 

0.254 

0.031 

0.044 

0.5  Selection 

Cutoff 

Selected 

Mean 

Measure  1 

Measure  2 

Measure  3 

Measure  4 

Measure  5 

Measure  6 

0.553 

0.549 

0.724 

0.684 

0.250 

0.789 

Non-selected 

Mean 

0.500 

0.212 

0.750 

0.251 

0.386 

0.659 

Selected 

Variance 

0.039 

0.174 

0.006 

0.163 

0.035 

0.043 

Non-selected 

Variance 

0.088 

0.115 

0.000 

0.142 

0.055 

0.091 

Denotes  a  weighted 
attribute  with  0  to  1 

Denotes  attribute  with 
zero  weight  assigned 

As  previously  shown,  when  the  bounds  for  the  weights  are  relaxed  to  0  to  1 , 
weight  is  assigned  to  those  measures  that  typically  have  the  greatest  difference  between 
selected  and  non-selected  means.  There  were  some  instances  where  there  was  a 
significant  difference  in  means  but  a  zero  weight  was  still  assigned.  This  usually 
occurred  when  the  variances  of  the  two  groups  was  large  enough  that  it  reduced  the 
effectiveness  of  the  differences  in  the  means  to  discriminate. 
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IV.D  Random  Data 


In  order  to  try  to  see  how  the  math  programming  and  diseriminant  analysis  would 


respond  to  different  alternative  sets,  a  random  set  of  42  alternatives  was  created  using 


random  values  between  0  and  1  for  14  attribute  measures.  The  random  values  for  the 


attributes  were  created  using  the  RAND  function  in  Microsoft  Excel.  The  original 


weights  of  the  MAJCOM  data  were  used  as  the  starting  weight  for  the  model  and  value 


score  cutoff  thresholds  of  0.4,  0.5  and  0.6  were  applied  to  determine  the  selection  groups. 


Table  9  shows  the  resulting  weights  obtained  when  allowed  to  vary  between  0  and  1. 


Table  9  Random  Alternative  Set  Weights  for  0  to  1  Bounds 


Selection 

Attribute  Weights 

Threshold  (#  Sel  / 

2 

3 

4 

5 

6 

7 

8 

9 

10 

11 

12 

13 

14 

0.4  (31/42) 

0.202 

0.124 

0 

0.585 

0.043 

0 

0 

0 

0 

0 

0 

0.036 

0.01 

0 

0.5  (19/42) 

0 

0 

0 

0.316 

0.38 

0 

0 

0 

0 

0 

0 

0.007 

0.297 

0 

Original  Weights 

0.06 

0.17 

0.08 

0.17 

0.17 

0.012 

0.03 

0.018 

0.06 

0.06578 

0.08211 

0.03289 

0.02461 

0.02461 

The  random  data  had  comparable  hit  rates  to  the  previous  examples  as  shown  in 
Table  10  below.  The  high  hit  rates  reflect  the  fact  that  discriminant  function  does  a  good 
job  of  classify  the  alternatives  into  selected  and  non-selected  groups. 

Table  10  Randon  Alternative  Set  Confusion  Matrix 


Random  DATA  0  to  1  Bound 


0.4 

Threshold 

(31/42) 

Select 

Non  Select 

Select 

31 

0 

Non  Select 

0 

11 

Hit  Rate 

100.0% 

0.5 

Threshold 

(19/42) 

Select 

Non  Select 

Select 

19 

0 

Non  Select 

1 

22 

Hit  Rate 

97.6% 
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Comparing  the  mean  of  the  selected  group  to  the  mean  of  the  non-selected  group 
and  the  variance  of  the  two  groups  yields  similar  results  as  the  MAJCOM  and  JIEDDO 
data. 

IV.E  Specific  Data  Set 

Finally,  a  very  specific  data  set  was  created  to  help  in  the  understanding  of  which 
attributes  were  getting  zero  weight  and  which  were  getting  a  high  proportion  of  the  global 
weight.  The  alternative  set  shown  in  Table  1 1  was  specifically  developed  in  an  attempt 
to  analyze  a  low  dimensional  problem  to  see  how  weight  was  being  applied. 


Table  1 1  Specific  Alternative  Data 


Measure  1 

Measure  2 

Measure  3 

Altl 

0.5 

0.6 

0.7 

Alt  2 

0.7 

0.6 

0.5 

Alt  3 

0.7 

0.1 

0.7 

Alt4 

0.5 

0.2 

0.5 

Alts 

0.4 

0.6 

0.4 

The  process  was  run  with  an  original  weighting  of  0.333  for  each  attribute 
measure  and  a  value  score  cutoff  threshold  of  0.6.  From  the  selection  threshold,  Alt  1 
and  Alt  2  belong  to  the  selected  group  and  Alts  3  through  5  are  in  the  non-selected  group. 
The  resultant  weighting  from  this  discriminant  analysis  and  compromise  programming 
yields  weights  of  0.003  for  Measure  1,  0.994  for  Measure  2  and  0.003  for  Measure  3. 

The  process  was  applied  a  second  time  with  an  original  weighting  of  0.4  for 
Measure  1,  0.2  for  Measure  2  and  0.4  for  Measure  3.  The  selection  threshold  was  again 
set  to  0.6.  This  criteria  assigned  Alt  1,  Alt  2  and  Alt  3  to  the  selected  group  and  Alts  4 
and  5  to  the  non-selected  group.  The  resultant  weighting  from  this  discriminant  analysis 
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and  compromise  programming  yields  weights  of  0.5  for  Measure  1,  0  for  Measure  2  and 
0.5  for  Measure  3.  The  elassifieation  hit  rate  for  both  runs  was  100%. 

IV.F  Example  of  Inconsistent  Decision 

In  order  to  ereate  an  ineonsistent  deeision,  the  MAJCOM  data  was  used  and  25 
alternatives  were  ehosen  to  be  in  the  seleeted  group,  leaving  17  in  the  non-seleeted  group. 
The  alternatives  were  distributed  sueh  that  there  were  alternatives  with  high  value  and 
low  value  scores  in  both  the  seleeted  and  non-seleeted  sets.  The  confusion  matrix  in 
Table  12  shows  the  impact  of  this  inconsistent  behavior. 

Table  12  Ineonsistent  Deeision  Confusion  Matrix 


Select 

Non  Select 

Select 

20 

5 

Non  Select 

7 

10 

Hit  Rate 

71.43% 

This  poor  hit  rate  demonstrates  that  the  deeision  made  was  ineonsistent  with  the 
information  captured  by  the  alternative  attribute  data.  A  decision  such  as  this  could 
illustrate  that  the  VFT  model  is  missing  some  attribute  or  information  that  is  important  to 
the  deeision  maker  or  that  there  are  higher  order  effeets,  non-linear  effeets,  or  interaetions 
that  could  not  be  accurately  captured  by  a  linear  diseriminant  model. 

IV.G  Summary 

This  chapter  provided  a  clear  and  concise  applieation  of  the  methodologies 
presented  in  ehapter  3  to  several  sets  of  alternative  data  and  VFT  models.  We 
demonstrated  an  ability  to  generate  an  optimized  diseriminate  function  that  can  correctly 
classify  alternatives  into  selected  and  non-selected  groups  and  provide  insight  into  whieh 
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attribute  measures  are  actually  the  most  discriminating  for  the  given  set  of  alternatives. 
Given  a  new  alternative  set  for  any  of  the  given  VFT  models  we  can  accurately  predict  if 
it  would  be  assigned  to  the  selected  or  non-selected  groups. 
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V.  Conclusions  and  Recommendations 


V,A  Introduction 

Air  Force  decision  makers  face  difficult  decisions  that  can  have  significant  impact 
to  future  force  capabilities  and  structure.  Often  these  decisions  are  made  more  difficult 
by  the  fact  that  the  goals  of  the  decision  are  often  conflicting.  VFT  models  provide  a  tool 
that  allows  analysts  to  help  the  DM  make  tradeoffs  between  conflicting  goals.  A  DM 
needs  to  have  a  detailed,  robust  and  mathematically  rigorous  model  to  aid  in  their 
decision  making.  These  models  should  be  validated  in  order  to  ensure  that  the 
mathematical  rigor  is  sound  and  accurate.  Also,  any  additional  insight  provided  to  the 
DM  can  better  prepare  them  for  the  next  time  the  VFT  model  may  be  used. 

V,B  Research  Contributions 

The  goal  of  this  research  was  to  contribute  to  the  fields  of  Decision  Analysis, 
specifically  Value  Focused  Thinking,  by  providing  a  technique  to  validate  models  and 
provide  insights  to  alternative  sets.  This  research  combined  Discriminant  Analysis  and 
Compromise  Programming  techniques  to  provide  insights  into  the  decision  made. 

The  first  contribution  is  in  the  area  of  VFT  model  validation.  This  research 
applied  a  two  group  discriminant  analysis  technique  to  statistically  generate  a 
classification  function  that  can  accurately  predict  assign  of  alternatives  based  upon  any 
VFT  model  and  alternative  set  provided. 

The  second  contribution  is  in  the  area  of  tradespace  or  sensitivity  analysis.  By 
allowing  the  discriminant  function  and  compromise  programming  to  find  a  set  of  weights 
with  a  0  to  1  bound,  we  can  see  what  attribute  measures  are  the  discriminating  factors. 
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The  measures  with  the  most  statistieal  weight  assigned  have  the  most  diseriminatory 
power  for  a  given  alternative  set  which  may  or  may  not  correspond  the  original  VFT 
weights.  If  an  attribute  measure  was  assigned  little  or  no  statistical  weight  it  means  that 
either  most  of  the  alternatives  scored  the  same,  or  the  means  of  the  selected  and  non- 
selected  groups  were  not  statistically  different  enough  to  contribute  to  discriminate  using 
that  attribute  measure.  Another  technique  to  provide  insight  into  alternative  sets  provided 
by  Dees  et  al  transforms  the  value  functions  to  amplify  the  differences  between 
alternatives  (Dees  et  al,  2010). 

V.C  Recommendations  for  Further  Research 

The  methodology  presented  in  this  research  applied  linear  discriminant  functions 
to  validate  the  VFT  models.  Future  Research  could  apply  other  multivariate  techniques 
such  as  higher  order  functions  or  neural  networks  if  a  linear  discriminant  model  indicates 
inconsistent  behavior.  This  could  account  for  any  interactions  between  attributes,  higher 
order  or  non-linear  effects. 

V,D  Conclusions 

Decision  Analysis  and  VFT  models  provide  robust,  reproducible  tools  to  DMs 
and  analysts  to  allow  complex  decisions  to  be  represented  in  an  understandable  way. 

VFT  models  are  widely  used  in  the  Air  Force  and  should  be  validated  as  often  as  feasible. 
This  research  demonstrated  a  sound  statistical  method  using  discriminant  analysis  and 
compromise  programming  to  valid  and  provides  insights  into  VFT  models  and  alternative 
sets. 
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Appendix  A:  MATLAB  Code  Example 

clc 

clear  all; 
close  all; 


%#  Tenets  Primary  Gap  Classification  Months 
%Suitability  Interop.  Issues  TRL  Months 
Capacity  Interaction  Min/Hr  Training  Hours 


Useful  Performance 
to  Fielding  %  Max 


ALT= 

[0 

.5 

1 

0  . 

.75 

1 

0 

1 

1 

0  . 

.  9 

0  . 

.7 

1 

1 

0. 

,75 

0  . 

.22 

0  . 

,75 

1 

0  . 

.5 

1 

1 

0  . 

,  9 

0  . 

,  6 

0  . 

,  912 

0. 

.75 

0  . 

.78 

0  . 

.75 

1 

0  . 

.25 

0  . 

.75 

0  . 

.5 

0  . 

.9  0 

0. 

.  918 

1 

0  . 

78 

0  . 

,75 

1 

0  . 

,25 

0  . 

.5 

0  . 

.5 

0  . 

,  93 

0  . 

,767 

0. 

.163 

0. 

,5 

0  . 

22 

0  . 

,75 

1 

0  . 

,25 

0  . 

.75 

0  . 

,5 

0  . 

,  15 

0 

0. 

.701 

0. 

.5 

1 

0  . 

.75 

0  . 

.187 

0  . 

.5 

0  . 

.5 

0  . 

.5 

0  . 

.  9 

0  . 

.7 

0  . 

.  912 

0 

0. 

,5 

1 

0  . 

.75 

0  . 

o 

LO 

,25 

0  . 

.5 

0  . 

.5 

0  . 

.2 

0  . 

,475 

0  . 

,  955 

0. 

.75 

0  . 

.33 

0  . 

,5  0. 

.023 

0  . 

.25 

1 

1 

0  . 

.  9 

0  . 

o 

CO 

0. 

,701 

0. 

.5 

1 

0  . 

.75 

0  . 

.187 

0  . 

.5 

0  . 

.75 

0  . 

.5 

0  . 

.3 

0  . 

.  6 

0  . 

.795 

0. 

.5 

1 

0  . 

.75 

0  . 

.5  0. 

.25 

0  . 

.5 

0  . 

.5 

0  . 

.  9 

0  . 

.  6 

0  . 

.795 

0 

0. 

,5 

0 

0  . 

.75 

1 

0  . 

o 

LO 

,75 

0  . 

.5 

0  . 

.2 

0  . 

,475 

0  . 

,  871 

0. 

.5 

1 

0  . 

.75 

0  . 

.5  0. 

.25 

0  . 

.5 

0  . 

.5 

0  . 

.  15 

0  . 

.733 

0 

0. 

,5 

1 

0  . 

.75 

0  . 

,023 

0  . 

,25 

0  . 

.75 

0  . 

,5 

0  . 

,3 

0  . 

,733 

0. 

.701 

0. 

.5 

0 

0  . 

.75 

0  . 

.074 

0  . 

.5 

1 

0  . 

.5 

0  . 

.  9 

0  . 

.867 

0  . 

.795 

0. 

,5 

0 

0  . 

.75 

0  . 

,187 

0  . 

,5 

1 

0  . 

.5 

0  . 

.3 

0  . 

,475 

0  . 

,396 

0. 

.5 

0  . 

44 

0  . 

.75 

1 

0 

1 

1 

0  . 

.3 

0  . 

.35 

1 

1 

0. 

,5 

0  . 

44 

0  . 

,75 

1 

0 

1 

1 

0 

0  . 

,35 

1 

1 

0. 

,5 

0 

0  . 

.5  1 

0 

1 

1 

0 

0  . 

.7 

1 

1 

1 

0 

0 

0  . 

.75 

1 

0 

1 

1 

0  . 

.2 

0  . 

.558 

1 

1 

1 

0. 

n 

,75 

0  . 

.22 

0  . 

,75 

1 

0  . 

.25 

0  . 

.75 

0  . 

,5 

0  . 

,9  0 

u 

0 

0  . 

22 

0  . 

,75 

0  . 

,074 

0  . 

.5 

1 

0  . 

.5 

0  . 

,  9 

0  . 

,475 

0 

0. 

,701 

0. 

.5 

0 

0  . 

.75 

0  . 

.187 

0  . 

.75 

0  . 

.5 

0 

0  . 

.  93 

0  . 

.867 

0.701 
?!  0.233 


0 . 912 


0 


57  0.912  1 

59  0.106 

0.039  0.596 

.2  0.617 


0.039 
0 . 163 
0 . 617 
55  0 
0.795 


0.5 

0 . 843 
0.241 
0 


1  0.918 

0.039  0.701 

1 
1 


0 . 617 


0 . 617 


0.918 

0.75 

0.233 

0.5  0.44 

1  0.78 

0.5 

0  0 

0.701 

0.75 

0 

0.5  0 
0.5  0 
0.5  0 


0 .89 
0 


0.75 


0 .187 


0.25 


0.75 

0.75 


0.023 

0.004 


0.5  ] 
0.25 


0 

0  . 


0.75 


0.023 


0.75 


0 


0 

0.75 

0.75 

0.75 


0.75 


0.25 


0.75 


0.074  0.25 

1  0.25 


Weights= [0.056 
0.056  0.087 


0.001 
0 . 176 


0 


0  . 

0.5  0 
0.75 


5  0. 
0 
0 


0.056 


0 .112 


0.75 

0.2  0. 

5  0  0 . 

0 . 15 

0  0 . 

5  0.1  0. 
0.35 
0  0 . 
0 .11 


0 


0 . 955 


0.2  0.433 


475  0.396  1 

1  0.558  0.955 


0 . 933 
1  0.35 


0.016 

0.396 


475  0.396  1 

0.795  0.039 

8  0  0.039  0] 

0.056  0.091 


0 . 617 
0 . 912 
1 

0.039 

1 

0 . 617 

0.701 

0.5 

0.037 


0.1  0.05] ; 


full=l 
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if  full 


1 


Scores=ALT*Weights ' ; 
popl=ALT (1:15,1:12); 
pop2=ALT (16 : 30,1:12); 

covp= (1/28) * ( (14*cov (popl) ) + (14*cov (pop2) ) ) 
b=inv (covp) * (mean (popl ) ' -mean (pop2 ) ' ) 
dscores=ALT ( : , 1 : 12 ) *b 

mid= . 5* (mean (popl ) ' -mean (pop2 ) ' ) ' *inv (covp) * (mean (popl ) ' +mean (pop2 ) ' ) 
tp=0 ; f n=0 ; tn=0 ; fp=0 ; 
for  i=l : 30 

if  dscores(i)  >=  mid  &  i  <=  15 
tp=tp+l 

elseif  dscores(i)<=  mid  &  i  <=  15 
fn=fn+l 

elseif  dscores(i)  <=  mid  &  i  >  15 
tn=tn+l 

else 

fp=fp+l 

end 

end 

CA=(tp+tn)/ (tp+tn+fn+fp) 

E=inv (covp) 

Ml=mean (popl) 

M2=mean (pop2) 
pop= [popl;pop2] 
for  i=l : 30 

num=exp ( ( - . 5 ) * (pop ( i , : ) -Ml ) *E* (pop ( i , : ) -Ml ) ' ) ; 
denom=num+exp ( ( - . 5 ) * (pop ( i , : ) -M2 ) *E* (pop ( i , : ) -M2 ) ' ) 
pp ( i ) =num/ denom 

end 

global  pop  pp 

p=12 

x0=ones (l,p) * (1/p) ; 
lb=Weights*0 . 1 ; 
ub=Weights* 1 . 1 ; 

Aeq=ones ( 1 , p) ; 
beq=l ; 

[x,  fval , exitf lag] =fmincon ( @globalfun,  xO  ,  [],  [ ] , Aeq, beq, lb,  ub) 
pred=pop*x ' 
else 

Scores=ALT*Weights ' ; 

popl=ALT (1:15,1:12); 
pop2=ALT (16 : 30,1:12); 

covp=  (1/28)  * ( (14*cov (popl) )  +  (14*cov (pop2) ) ) 
b=inv (covp) * (mean (popl ) ' -mean (pop2 ) ' ) 
dscores=ALT ( : , 1 : 12 ) *b 
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mid= . 5* (mean (popl ) ' -mean (pop2 ) ' ) ' *inv (covp) * (mean (popl ) ' +mean (pop2 ) ' ) 
tp=0 ; f n=0 ; tn=0 ; fp=0 ; 
for  i=l : 30 

if  dscores(i)  >=  mid  &  i  <=  15 
tp=tp+l 

elseif  dscores(i)<=  mid  &  i  <=  15 
fn=fn+l 

elseif  dscores(i)  <=  mid  &  i  >  15 
tn=tn+l 

else 

fp=fp+l 

end 

end 

CA=(tp+tn)/ (tp+tn+fn+fp) 

E=inv (covp) 

Ml=mean (popl) 

M2=mean (pop2) 
pop= [popl;pop2] 
for  i=l : 30 

num=exp ( ( - . 5 ) * (pop ( i , : ) -Ml ) *E* (pop ( i , : ) -Ml ) ' ) ; 
denom=num+exp ( ( - . 5 ) * (pop ( i ,  : ) -M2 ) *E* (pop ( i ,  : ) -M2 )  ' ) 
pp ( i ) =num/ denom 

end 

end 
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Appendix  B:  MAJCOM  Alternative  Set 

Table  13  MAJCOM  Alternative  Set 


Measure  1 

Measure  2 

Measure  3 

Measure  4 

Measure  5 

Measure  6 

Measure  7 

Measure  8 

Measure  9 

Measure  10 

Measure  11 

Measure  12 

Measure  13 

Measure  14 

Alternative  1 

1 

1 

0.994599525 

0.75 

0 

1 

1 

1 

0.66 

Alternative  2 

1 

1 

1 

0.9 

1 

0 

1 

0.8 

0.6 

0.66 

Alternative  3 

1 

1 

1 

0.65 

0 

1 

1 

0.8 

0.8 

0.666666667 

0.66 

Alternative  4 

1 

0.75 

1 

0.9 

1 

0 

0.5 

0.6 

1 

1 

Alternative  5 

0.75 

0.75 

0 

1 

0 

0 

0.666666667 

0.66 

Alternative  6 

1 

1 

0.5 

0.994599525 

0.65 

0 

1 

0.8 

0.8 

0.666666667 

0.66 

Alternative  7 

1 

1 

0.5 

0.985918162 

0.65 

0 

1 

0.8 

0.8 

0.666666667 

0.66 

Alternative  8 

1 

0.5 

0.97196271 

0.65 

0 

1 

0 

0.8 

0.8 

0.666666667 

0.66 

Alternative  9 

1 

0.7 

0.612501618 

1 

1 

0.75 

1 

1 

0.8 

0.4 

0.666666667 

0.66 

Alternative  10 

1 

0.75 

0.949529063 

0.65 

0 

1 

1 

0.8 

0.4 

0.666666667 

0.66 

Alternative  11 

0.5 

0.913466561 

0.65 

0 

1 

0.666666667 

0.8 

0.8 

0.666666667 

1 

Alternative  12 

0.7 

0.1 

0.97196271 

0.9 

0 

1 

1 

0.8 

0.8 

0.66 

Alternative  13 

0 

1 

0.9 

0 

1 

0 

1 

1 

0 

0.66 

Alternative  14 

0.7 

1 

1 

0.9 

0 

0 

1 

1 

0.2 

0.6 

0.333333333 

0.66 

Alternative  15 

1 

0.3 

0.27957697 

0.9 

1 

0.75 

1 

0 

0.666666667 

1 

Alternative  16 

1 

1 

0.18674946 

0.9 

1 

0.75 

0.666666667 

0.8 

0 

0.666666667 

0.66 

Alternative  17 

1 

1 

0.949529063 

0.5 

0 

1 

0.75 

0 

0.666666667 

0 

0.666666667 

0.33 

Alternative  18 

1 

0.75 

0 

0.985918162 

0.9 

1 

1 

0 

0 

0.666666667 

0.66 

Alternative  19 

1 

0.7 

0.7 

0 

0 

0 

0 

0.8 

1 

1 

0.33 

Alternative  20 

1 

0.7 

0 

0.855495422 

0.9 

1 

1 

0.75 

0.5 

0 

0.666666667 

0.66 

Alternative  21 

1 

0 

0.994599525 

0.75 

1 

0 

0.75 

0.666666667 

0.5 

0 

0.666666667 

0.66 

Alternative  22 

0.5 

0.75 

0.994599525 

0.7 

0 

0 

0 

1 

0.8 

0.4 

0.666666667 

0.66 

Alternative  23 

1 

0.75 

0 

0.9 

0 

0 

0.75 

0 

1 

0.8 

0.6 

1 

0.66 

Alternative  24 

0.75 

1 

0.123269143 

0.9 

1 

0.75 

1 

0.666666667 

1 

0 

0.333333333 

1 

Alternative  25 

0.75 

0.3 

0.415318992 

1 

0.75 

0 

1 

1 

0.4 

0.666666667 

0.66 

Alternative  26 

0.75 

1 

0.985918162 

0.9 

0 

0 

0.333333333 

0.1 

0 

0.333333333 

1 

Alternative  27 

0.75 

0.75 

1 

0.612501618 

0.65 

0 

0 

0 

1 

0.8 

0.6 

0.66 

Alternative  28 

1 

0 

0.762305745 

0.5 

0 

1 

0.75 

0 

1 

0.8 

0.6 

0.66 

Alternative  29 

0.75 

0.7 

0.3 

0.612501618 

0.65 

0 

1 

0 

0.8 

0.4 

0.666666667 

1 

Alternative  30 

0.75 

0.7 

0.3 

0.762305745 

0.5 

0 

1 

0.75 

0.666666667 

0 

0.666666667 

0.66 

Alternative  31 

1 

0.7 

0 

0.006492522 

0.9 

1 

1 

0.75 

0 

0.666666667 

1 

1 

1 

Alternative  32 

1 

1 

0 

0.27957697 

0.5 

0 

1 

0 

0 

0.8 

1 

0.666666667 

0.66 

Alternative  33 

1 

0.7 

0.5 

0.415318992 

0.9 

1 

0 

0 

0.333333333 

0.1 

0 

0.333333333 

1 

Alternative  34 

0.75 

0.7 

0.5 

0.18674946 

0.75 

0 

0 

0 

0.666666667 

0.8 

0.6 

0.666666667 

0.33 

Alternative  35 

0.5 

0.75 

0 

0.913466561 

0.5 

1 

0 

0 

0 

0.5 

0.6 

0.666666667 

0.66 

Alternative  36 

0.75 

0.75 

0.5 

0.015986561 

0.7 

1 

0 

0 

0 

1 

0.8 

1 

1 

0.66 

Alternative  37 

0.75 

0 

0.02986973 

0.75 

0 

0 

0 

1 

0.8 

0.8 

0.33 

Alternative  38 

0 

0.079857975 

0.7 

0 

0 

0 

0.666666667 

0.8 

0 

0.666666667 

0.66 

Alternative  39 

0 

0.123269143 

0.5 

0 

1 

0 

0.333333333 

0.8 

0.6 

0.666666667 

1 

Alternative  40 

0.75 

0.7 

0 

0.050171142 

0.7 

0 

0 

0 

1 

1 

0 

1 

Alternative  41 

0.7 

0 

1 

0 

1 

0 

0 

0.333333333 

0 

0 

0.333333333 

1 

Alternative  42 

0 

0 

0 

1 

0 

1 

0 

0 

0.8 

0 

0.333333333 

0.66 
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Appendix  C:  MAJCOM  Means  and  Variances  Table 


Table  14  MAJCOM  Means  and  Variances  Table 


0.5  Selection 

Cutoff 

Measure  1 

Measure  2 

Measure  3  Measure  4  Measure  5  Measure  6  Measure  7  Measure  8 

Measure  9  Measure  10 

Measure  11  Measure  12 

Measure  13 

Measure  14 

Selected  Mean 

0.931 

0.836 

0.495 

0.675 

0.758  0.450 

0.775 

0.450 

0.550 

0.875 

0.775  0.450 

0.742 

0.712 

Non-selected 

Mean 

0.500 

0.350 

0.000 

1.000 

0.000  1.000 

1.000 

0.000 

0.000 

0.667 

0.400  0.000 

0.333 

0.830 

Selected 

Variance 

0.019 

0.019 

0.188 

0.144 

0.025  0.254 

0.179 

0.196 

0.254 

0.044 

0.054  0.139 

0.043 

0.038 

Non-selected 

Variance 

0.500 

0.245 

0.000 

0.000 

0.000  0.000 

0.000 

0.000 

0.000 

0.222 

0.320  0.000 

0.000 

0.058 

O.SSelection 

Cutoff 

Measure  1 

Measure  2 

Measure  3  Measure  4  Measure  5  Measures  Measure  7  Measures 

Measure  9  Measure  10 

Measure  11  Measure  12 

Measure  13 

Measure  14 

Selected  Mean 

0.950 

0.847 

0.610 

0.831 

0.780  0.467 

0.833 

0.542 

0.667 

0.911 

0.787  0.413 

0.733 

0.706 

Non-selected 

Mean 

0.813 

0.729 

0.125 

0.342 

0.575  0.500 

0.667 

0.146 

0.167 

0.750 

0.683  0.467 

0.694 

0.747 

Selected 

Variance 

0.015 

0.019 

0.180 

0.070 

0.024  0.257 

0.144 

0.186 

0.230 

0.030 

0.051  0.127 

0.041 

0.029 

Non-selected 

Variance 

0.092 

0.069 

0.051 

0.159 

0.091  0.273 

0.242 

0.119 

0.152 

0.083 

0.103  0.192 

0.070 

0.064 

0.7  Selection 

Cutoff 

Measure  1 

Measure  2 

Measures  Measure  4  Measure  5  Measures  Measure  7  Measures 

Measure  9  Measure  10 

Measure  11  Measure  12 

Measure  13 

Measure  14 

Selected  Mean 

0.970 

0.860 

0.628 

0.847 

0.808  0.520 

0.800 

0.590 

0.720 

0.933 

0.804  0.432 

0.733 

0.688 

Non-selected 

Mean 

0.824 

0.744 

0.241 

0.461 

0.594  0.412 

0.765 

0.191 

0.235 

0.765 

0.688  0.424 

0.706 

0.761 

Selected 

Variance 

0.012 

0.019 

0.180 

0.079 

0.020  0.260 

0.167 

0.182 

0.210 

0.019 

0.040  0.136 

0.037 

0.028 

Non-selected 

Variance 

0.068 

0.052 

0.121 

0.151 

0.070  0.257 

0.191 

0.129 

0.191 

0.080 

0.101  0.159 

0.068 

0.053 

O.SSelection 

Cutoff 

Measure  1 

Measure  2 

Measures  Measure  4  Measures  Measures  Measure  7  Measures 

Measure  9  Measure  10 

Measure  11  Measure  12 

Measure  13 

Measure  14 

Selected  Mean 

0.979 

0.888 

0.758 

0.950 

0.779  0.417 

1.000 

0.479 

1.000 

0.972 

0.808  0.650 

0.778 

0.717 

Non-selected 

Mean 

0.883 

0.783 

0.357 

0.587 

0.698  0.500 

0.700 

0.408 

0.333 

0.822 

0.737  0.340 

0.700 

0.718 

Selected 

Variance 

0.005 

0.020 

0.101 

0.012 

0.022  0.265 

0.000 

0.255 

0.000 

0.009 

0.015  0.074 

0.027 

0.018 

Non-selected 

Variance 

0.051 

0.039 

0.183 

0.157 

0.061  0.259 

0.217 

0.179 

0.230 

0.059 

0.086  0.144 

0.056 

0.047 

Column 

Variance 

0.039 

0.035 

0.190 

0.142 

0.050  0.256 

0.172 

0.196 

0.256 

0.049 

0.066  0.142 

0.048 

0.038 

Column  Mean 

0.911 

0.813 

0.471 

0.691 

0.721  0.476 

0.786 

0.429 

0.524 

0.865 

0.757  0.429 

0.722 

0.718 

Denotes  a  weighted 
attribute  with  0 to  1  bound 

Denotes  attribute  with 
zero  weight  assigned 
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Appendix  D:  JIEDDO  Alternative  Set 

Table  15  JIEDDO  Alternative  Set 


Alternative 

#Tenets 

Primary  Gap  Classification 

Ndonths  Useful 

Performance  Suitability 

Interop.  Issues 

TRL 

Months  to  Fielding 

%  Max  Capacity 

Interaction  Min/Hr  Training  Hours  Training  Level 

Value 

DD 

0.5 

1 

0.75 

1 

0 

1 

1 

0.9 

0.7 

1 

1 

1 

0.6 

0.8223 

BB 

0.75 

0.22 

0.75 

1 

0.5 

1 

1 

0.9 

0.6 

0.912 

1 

0.701 

0.6 

0.72467 

F 

0.5 

0.44 

0.75 

1 

0 

1 

1 

0.3 

0.35 

1 

1 

1 

0.6 

0.68194 

CC 

0.75 

0.78 

0.75 

1 

0.25 

0.75 

0.5 

0.9 

0.6 

0.871 

0.233 

0.918 

0.6 

0.676251 

E 

0.5 

0.44 

0.75 

1 

0 

1 

1 

0 

0.35 

1 

1 

1 

0.6 

0.67084 

AA 

1 

0.78 

0.75 

1 

0.25 

0.5 

0.5 

0.93 

0.767 

0.912 

0 

0.163 

0.6 

0.632792 

Z 

0.5 

0.22 

0.75 

1 

0.25 

0.75 

0.5 

0.15 

0.867 

0.912 

1 

0.701 

0.6 

0.610872 

J 

0.5 

0 

0.5 

1 

0 

1 

1 

0 

0.7 

1 

1 

1 

0.6 

0.599 

B 

0 

0 

0.75 

1 

0 

1 

1 

0.2 

0.558 

1 

1 

1 

0.6 

0.584448 

R 

0.5 

1 

0.75 

0.187 

0.5 

0.5 

0.5 

0.9 

0.7 

0.912 

0.039 

0.106 

0.6 

0.575637 

T 

0.5 

1 

0.75 

0.5 

0.25 

0.5 

0.5 

0.2 

0.475 

0.955 

0.039 

0.596 

0.6 

0.573493 

X 

0.75 

0.33 

0.5 

0.023 

0.25 

1 

1 

0.9 

0.8 

0.912 

0.617 

0.701 

0.6 

0.570985 

Y 

0.5 

1 

0.75 

0.187 

0.5 

0.75 

0.5 

0.3 

0.6 

0.795 

0.039 

0.5 

0.6 

0.569837 

S 

0.5 

1 

0.75 

0.5 

0.25 

0.5 

0.5 

0.9 

0.6 

0.795 

0 

0.163 

0.6 

0.56535 

P 

0.75 

0.22 

0.75 

1 

0.25 

0.75 

0.5 

0.9 

0.8 

0.631 

0.617 

0 

0.6 

0.552399 

w 

0.5 

0 

0.75 

1 

0.5 

0.75 

0.5 

0.2 

0.475 

0.871 

0.617 

0.843 

0.6 

0.549229 

D 

0.5 

1 

0.75 

0.5 

0.25 

0.5 

0.5 

0.15 

0.733 

0.795 

0 

0.241 

0.6 

0.548948 

C 

0.5 

1 

0.75 

0.023 

0.25 

0.75 

0.5 

0.3 

0.733 

0.795 

0 

0.701 

0.6 

0.538074 

U 

0.5 

0 

0.75 

0.074 

0.5 

1 

0.5 

0.9 

0.867 

0.795 

1 

0.918 

0.6 

0.53684 

L 

0 

0.22 

0.75 

0.074 

0.5 

1 

0.5 

0.9 

0.475 

0.955 

0.617 

0.701 

0.6 

0.497437 

1 

0.5 

0 

0.75 

0.187 

0.75 

0.5 

0 

0.93 

0.867 

0.955 

0.617 

0.918 

0.6 

0.487285 

Q 

0.75 

0.89 

0.75 

0.187 

0.25 

0.75 

0 

0.2 

0.433 

0.912 

0.233 

0 

0.6 

0.482003 

G 

0.5 

0.44 

0.75 

0.023 

0.5 

1 

0 

0.2 

0.475 

0.396 

1 

1 

0.6 

0.479416 

0 

1 

0.78 

0.75 

0.004 

0.25 

0.5 

0 

0.1 

0.558 

0.955 

0.039 

0.5 

0.6 

0.457869 

N 

0 

0 

0.75 

0.023 

0.75 

0 

1 

0.15 

0.933 

0.016 

1 

0.701 

0.6 

0.407324 

A 

0.75 

0 

0.75 

1 

0.25 

0.75 

0 

0.1 

0.35 

0.396 

0.617 

0 

0.6 

0.389879 

K 

0.5 

0 

0.75 

0.074 

0.25 

0.5 

0.5 

0.1 

0.475 

0.396 

1 

0.701 

0.6 

0.379038 

M 

0.5 

0 

0.75 

1 

0.25 

0.5 

0 

0 

0.35 

0.795 

0.039 

0.5 

0.6 

0.372793 

V 

0.5 

0 

0.75 

0.187 

0.5 

1 

0.5 

0.3 

0.475 

0.396 

0.039 

0.701 

0.6 

0.370987 

H 

0.5 

0 

0.75 

0.001 

0 

0.75 

0 

0 

0.8 

0 

0.039 

0 

0.6 

0.168105 
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Appendix  E:  JIEDDO  Means  and  Variances  Table 


Table  16  JIEDDO  Means  and  Variances  Table 


0.4  Selection 

Cutoff 

Measure  1 

Measure  2 

Measure  3 

Measure  4 

Measure  5 

Measure  6 

Measure  7 

Measure  8 

Measure  9 

Measure  10 

Measure  11 

Measure  12 

Selected 

Mean 

0.530 

0.510 

0.730 

0.540 

0.310 

0.750 

0.580 

0.496 

0.641 

0.842 

0.548 

0.643 

Non-selected 

Mean 

0.550 

0.000 

0.750 

0.452 

0.250 

0.700 

0.200 

0.100 

0.490 

0.397 

0.347 

0.380 

Selected 

Variance 

0.064 

0.166 

0.005 

0.193 

0.048 

0.068 

0.118 

0.142 

0.028 

0.048 

0.188 

0.118 

Non-selected 

Variance 

0.013 

0.000 

0.000 

0.254 

0.031 

0.044 

0.075 

0.015 

0.034 

0.079 

0.196 

0.127 

0.5  Selection 

Cutoff 

Measure  1 

Measure  2 

Measure  3 

Measure  4 

Measure  5 

Measure  6 

Measure  7 

Measure  8 

Measure  9 

Measure  10 

Measure  11 

Measure  12 

Selected 

Mean 

0.553 

0.549 

0.724 

0.684 

0.250 

0.789 

0.684 

0.523 

0.646 

0.888 

0.537 

0.645 

Non-selected 

Mean 

0.500 

0.212 

0.750 

0.251 

0.386 

0.659 

0.227 

0.271 

0.563 

0.561 

0.476 

0.520 

Selected 

Variance 

0.039 

0.174 

0.006 

0.163 

0.035 

0.043 

0.061 

0.144 

0.024 

0.010 

0.207 

0.122 

Non-selected 

Variance 

0.088 

0.115 

0.000 

0.142 

0.055 

0.091 

0.118 

0.109 

0.042 

0.136 

0.171 

0.133 

0.6Selection 

Cutoff 

Measure  1 

Measure  2 

Measure  3 

Measure  4 

Measure  5 

Measure  6 

Measure  7 

Measure  8 

Measure  9 

Measure  10 

Measure  11 

Measure  12 

Selected 

Mean 

0.643 

0.554 

0.750 

1.000 

0.179 

0.857 

0.786 

0.583 

0.605 

0.944 

0.748 

0.783 

Non-selected 

Mean 

0.500 

0.386 

0.728 

0.381 

0.337 

0.707 

0.435 

0.384 

0.619 

0.714 

0.444 

0.543 

Selected 

Variance 

0.039 

0.092 

0.000 

0.000 

0.036 

0.039 

0.071 

0.172 

0.039 

0.003 

0.190 

0.093 

Non-selected 

Variance 

0.057 

0.199 

0.005 

0.164 

0.043 

0.066 

0.121 

0.131 

0.030 

0.090 

0.174 

0.126 

JIEDDO 

Selection 

Measure  1 

Measure  2 

Measure  3 

Measure  4 

Measure  5 

Measure  6 

Measure  7 

Measure  8 

Measure  9 

Measure  10 

Measure  11 

Measure  12 

Cutoff 

Selected 

Mean 

0.583 

0.622 

0.733 

0.545 

0.333 

0.750 

0.600 

0.589 

0.666 

0.842 

0.375 

0.597 

Non-selected 

Mean 

0.483 

0.229 

0.733 

0.505 

0.267 

0.733 

0.433 

0.272 

0.565 

0.694 

0.655 

0.601 

Selected 

Variance 

0.022 

0.170 

0.004 

0.171 

0.029 

0.045 

0.043 

0.130 

0.030 

0.021 

0.194 

0.099 

Non-selected 

Variance 

0.099 

0.099 

0.005 

0.224 

0.064 

0.085 

0.183 

0.130 

0.040 

0.136 

0.157 

0.169 

Denotes  a  weighted 
attribute  with  0  to  1 

Denotes  attribute  with 
zero  weight  assigned 
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Appendix  F:  Blue  Dart 


The  Air  Force  decision  makers  have  to  make  billion  dollar  decisions  every  day. 
Analysts  provide  insights  to  the  decision  maker  using  decision  analysis  tools  and  models. 
Often  these  models  that  are  used  to  make  complicated  decisions  are  not  validated  using 
mathematical  or  statistical  techniques.  This  research  provides  a  means  to  mathematically 
validate  a  value  focused  thinking  decision  model  for  a  given  set  of  alternatives  that  were 
considered.  Additionally  it  provides  insight  into  what  aspects  or  attributes  of  the 
alternatives  were  the  key  factors  that  helped  make  the  decision. 
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Appendix  G:  Quad  Chart 
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